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SELECTING EDUCATIONAL RESEARCHERS AND EVALUATORS 

Ja son Mi llman 

ABSTRACT 

Aimej at those individuals who are in a position to hire or 
promote educational researchers or evaluators, this paper 
provides some practical suggestions for assessing these 
personnel* Me ta-evalua tors (evaluators who evaluate evaluations) 
and referees of research proposals will also find this article 
relevant* Selection of a research or evaluation (R and E) firm 
is not treated separately from the task of hiring an individual; 
the quality of work done by a firm depends largely on the people 
who do the job. Much consideration should be given to specifying 
job descriptions and requirements* The value of R and E 
competencies depends upon the specific tasks expected to-be 
^ ^ ^ performed. A synthesis of the efforts of a task force of the 
ig|o«S American Educational Research Association to identify educational 
L,u,«. ^oooT^hJ; R and E competencies grouped under 25 general tasks is included 
mSS w>gg5|5 in the document. In an effort to identify a universe of 
iuJgg^ulyS J^g evaluation competencies, Stufflebeam and Bunda produced 
J5z5J!25iS|o approximately 250 items grouped under eight major categories. 
^S^oz!il?Hzs The categories and examples of corresponding self assessment 
jSi^s^zi^^l items are also included. These groups of competencies reference 
3; g'^itftp tasks, abilities, and knowledges of the almost total exclusion of 
^S^gSfnS sensitivities. Interpersonal sensitivities and personality 
w characteristics may be even more important than technical 
knowledges and skills in insuring the success of R and E. 
Several strategies for assessing whether an individual possesses 
the competencies needed for a specific job are considered. These 
include discussions of certification, formal training, testing, R 
and E output, bibliographic and academic characteristics, and 
membership in special professional associations and directories 
having more stringent entry requirements than presently exist. 



In t roduc t ion 

This paper provides some practical suggestions for selecting 
educational researchers and evaluators. It is directed primarily to 
those ind ividuals who are in a position to hire (or possibly to 
promote) educational researchers or evaluators. Me ta-evalua tor s 
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(th'.t 4s, evaluators who evaluate evaluations) and referees of 
research proposals and reports will find this article of some 
r elevance« 

Personnel selection and promotion is a serious matter* 
Decisions about applicants and employees have, of course, direct and 
long-lasting effects on them and their employers. The work of the 
educational researcher and evaluator is f ar--reaching, touching in 
profound ways the lives of those people who have a vested interest 
in the programs under study* Thus, the assessment of educational' 
researchers and evaluators merits special care* 

Many people have provided extensive and thoughtful documents 
that define educational research and evaluation (R and E) 
components, chat state criteria in judging R and E products, and 
offer suggestions and materials for the training of researchers and 
evaluators* Much of this work will be cited* The biases of the 
author, however, cannot help but enter into the choice of items that 
were referenced and the application of this material to the 
personnel-^selection problem. Thus, what follows is more than a 
review; it is a point of view* 

The selection of an R or E firm will not be treated separately 
from the task of hiring an individual* The assumption being made 
here is that the quality of the work of a firm depends to a large 
extent on the people who are to do the job. Prestigious 
institutions often produce shoddy work when tasks are turned over to 
incapable staff or to individuals who have been frantically 
recruited after the job is awarded* The contractor should insist on 
knowing who within the organization will be responsible for the 
conceptualization and conduct of the major phases of the inquiry and 
how much time such people will commit to the project. 

Although it is true that much attention has heen given to 
definitions of, and distinctions between, R and E (See, for example, 
29, 8, and 9 and, more recently 10 and 26), they will be treated 
together in this paper* Although R and E can be clearly 
differentiated at a conceptual level, when attention is turned to 
needed ski lis and competencies, there are more similarities than 
differences. Moreover, it is probably fair to say that the 
importance of any given skill depends less on whether a position is 
labeled R or E than on the specific job requirements* 

Job Description and Requirement s 

Most people would agree that an appropriate starting place for 
selecting personnel is with a job description, including a statement 
of the job requirements Assessment always involves some criteria, 
and it is preferable to make these expectations explicits 

Recently, the author was involved with evaluation of a large 
federal program* Part of the evaluation task required an extensive 
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amount of interviewing* It was not enough for us to hire an 
individual with interviewing skills* The population being 
interviewed was rather special, and it was felt that only 
interviewers with particular racial and cultural characteristics 
could be maximally effective* This incident points up the 
importance of considering not only the competencies and skills of 
the applicant but also other characteristics that can influence the 
effectiveness of the person in the job* These characteristics are 
likely to be specific to, and rather obvious in, a given context* 



R and E Competencies 

Not all R and E competencies are equally important* As mentioned 
above, their value depends upon the specific tasks that the 
individual will be expected to perforin* That is why it was stated 
above that much consideration should be given, to specifying job 
descriptions and requirements* Nevertheless, extant lists of R and 
E skills can suggest to employers qualifications appropriate for 
their situations* References to such liots are provided below* 

A caveat to remember about R and E competencies is that all 
skills are not needed initially, nor are they equally difficult to 
acquire* An applicant may have qualities that offset initial 
deficiencies* Asked what he would look for in hiring an evaluator, 
one illustrious educational researcher-evaluator remarked that he 
would require the applicant to be smart and have statistical and 
psychometric skills* He argued that with a good mind, the applicant 
could easily pick up information about evaluation theories and 
strategies* An inventory of R and E skills should not be considered 
a catalog of required entry competencies* 

Because of their utility in the training of R and E personnel, 
statements of R and E skills can be found in the documentation of 
training programs (12)* 

A task force on research training sponsored by the American 
Educational Research Association (AERA) produced a list of 
competencies based on a logical analysis of research and evaluation 
tasks (9, 25)* Later empirical work by several investigators 
involved either asking respondents to rate the importance of 
competencies listed a priori (3, 15) or using task analysis 
interview techniques (1, 16)* 

A synthesis of the educational R and E competencies identified 
by the AERA task force has recently appeared (23)* The competencies 
are grouped under the 25 general R and E tasks identified in Table 
1* The specific competencies have not been reproduced here* 
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Table 



1 



General Research and Evaluation Tasks* 



1« Obtaining information about an area to be researched or a 
phenomenon to be evaluated. 

2« Drawing implications from results of prior research and 
practice. 

3. Conceptualizing the research problem or defining the object of 

the evaluation. 

4. Selecting an appropriate inquiry strategy for addressing the 

research or evaluation problem. 

5. Formulating hypotheses or questions to be answered by the 

study . 

6* Specifying data or evidence necessary for a rigorous test of 
the hypothesis or an unequivocal answer to the research or 
evaluation question. 

7. Selecting appropriate research and evaluation designs to 

collect data to test the hypothesis or answer the question. 

8* Identifying the population to which results should be 

generalized, and selecting a sample of the population. 

9. Applying the research or evaluation design and recognizing or 
controlling threats to validity. 

10. Identifying at appropriate levels of generality the goals of 
the program to be evaluated. 

11« Assessing the value and feasibility of program goals. 

12. Identifying standards or norms for judging worth of the 

phenomenon to be evaluated. 

13. Translating broad objectives into specific (measurable) 

ob j ec tives . 
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14. Identifying classes of variables for measurement. 

15. Selecting or developing techniques of measurement. 

16. Assessing the validity of measurement techniques. 

17. Using appropriate methods to collect data (tests, interviews, 

unobtrusive measures, etc.). 

18. Monitoring the program to detect deviations from design or 

specified procedures. 

19. Choosing and employing appropriate techniques of statistical 

analysis . 

20. Using electronic computers and computer-^r elated equipment. 

21. Interpreting and drawing appropriate conclusions from data 

analysis . 

22. Reporting research and evaluation findings and implications. 

23. Making recommendations as a result of the evaluation. 

24. Providing immediate feedback on program performance for use in 

decisions about program modification. 

25. Obtaining and managing resources (material and human) necessary 

to conduct the research or evaluation study. 



* From B. R. Worthen, Competencies for Educational Research and 
Evaluation (23). 



Coller (4) has classified evaluation tasks into 16 general 
groupings. In an effort to define a universe of evaluation 
competencies, Stufflebeam (20) and Bunda (3) produced approximately 
250 items for a self-assessment of evaluation skills. The items 
were grouped under eight major categories. These categories and 
examples of c orrc '=?pond ing sel f --assessment items are given in Table 
2. 
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Table 2 



Examples of Evalua tion Competencies 
Organized into Eight Categories* 

1* Knowledge of Innovation in Evaluation 

* I can compare and contrast instructional research and 
evaluation • 

* I can select, organize, and lead groups of professionals to 
generate alternative program strategies and project designs* 

2. Public Relations 

* I can use audio-^visual aids appropriately in making oral 
evaluation reports • V 

* I can design an evaluation report to be presented by 
video- tape . 

3* Data Processing 

* I can design and develop a data bank. 

* I can develop forms and procedures for managing the data 
processing operations of an evaluation system. 

4. Educational Measurement 

* I can write f orced--choice and free-choice test items. 

* I can select a test that has been normed on an appropriate 
group . 

5« Evaluation Administration 

* X can marshall political support for evaluation activities. 

* I can delineate evaluation authority and re^^ponsib il i ty 
within the agency's organizational structure. 

6. Relating Evaluation to Relevant Disciplines 

* I can write an analysis of the relevance of economic theory 
to educational evalua tion. 

* I can state the basic principles of value theory and utility 
theory and compare and contrast the relevance of these fields 
for theory development in educational evaluation. 

7. Communications 

* I can prepare a 20-minute slide-tape presentation on problems 
in evalua tion . 
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* I can organize and administer an editorial service in 
relation to evaluation reports* 



8« Research Design Analysis 

* I can design controlled research studies of small units in a 
curriculum « 

* I can defend the choice of an oblique rotation or an 
orthogonal rotation in a factor analysis study. 



* From M. A. Bunda, A Partial Validation of the New 
Concept ualization of Evaluation Competencies (3) and 
Stufflebeam, A New Conceptualization of Evaluation 
Competencies (20) « 



Schalock et al * (16) considered tasks and enablers as two 
facets of work r equir ement s • Tasks are involved in producing an 
output to its specified standard; enablers are relied upon or 
utilized in the production of an output* Enablers are of three 
types: knowledges (facts, general i za ' ions , principles), skills 
(abilities, proficiencies, expertnesses) , and sensitivities 
(awarenesses, attitudes, personalities)* In spite of the similarity 
between tasks (for example, administer data instruments) and skills 
(for example, ability to administer data instruments), there does 
seem to be value in distinguishing between the tasks that a worker 
will need to perform and the knowledges, skills, and sensitivities 
purportedly required to complete the task* 

The groups of competencies in the accompanying tables reference 
tasks, abilities, and knowledges to the almost total exclusion of 
sensitivities* It may be that interpersonal sensitivities and 
personality characteristics are even more important than technical 
knowledges and skills in insuring the success of researchers and, 
especially, of evaluators* (See 27 for an illustration of how such a 
case might be made*) Evaluators and researchers, for example, must 
marshall support, attain credibility, facilitate interactions, and 
communicate effectively* Responsive evaluation (18) yields 
information audiences want* This requires a person who is aware of 
the needs of others and capable of working with groups of people* 
Because of the human element involved in R and E activities, the 
administrator should not be loath to consider personality 
characteristics when judging the suitability or pronise of an 
applicant or employee* 
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Sources of Evidei 



About Competence 



It is one thing to entify important R and E competencies but quite 
another to evaluate hether a given individual possesses those 
skills needed in a pc. ticular job. This section of the review 
contains a discussion f several strategies for making this 
assessment « 



The literature on certification is relevant in this regard. 
For at least 10 years (see, for example, 5), there have been 
suggestions that educational R and E personnel be certified^ 
Worthen (22) has offered a detailed discussion of issues and 
procedures in the certification of educational evaluators that also 
has relevance for researchers. Many of the ideas that follow have 
been taken from that reference. 



Formal Training: This source of evidence of competence has a long 
history and until recently has been largely uncritically accepted as 
a valid indicator. Formal training was recommended by Di Lorenzo 
(5) as the basis for certification. Detailed accounting of 
desirable cour a work and other training experiences are often 
included in descriptions of evaluation training programs (see 6 for 
a summary). Training requirements for educational researchers have 
been prepared by a committee of the Council of Europe and are 
reported by Harnqvist (11). 

One should keep in mind that completion of course work is no 
guarantee that the individual will have the enablers needed to carry 
out R and E tasks. Nevertheless, training and experience do have 
some merit, since they are certain to have had some influence on the 
apprentice . 

Tests: A more direct measure of competence can be obtained from 
tests of R and E skills. Some self-report measures are available 
(see 3 and 24) but, on the face of it, are suspect as selectjors 
because of the applicant's desire to appear to be qualified. There 
is some evidence (14), however, that applicants seeking new 
positions do not over-rate their abilities on these self-report 
devic es . 



One serious attempt to construct a formal test of R and E 
skills (28) lacks normative data that would be needed for meaningful 
interpretation of test results. A collection of test items in 
evaluation is currently being assembled by Daniel Stufflebeam 
(personal Correspondence), but these items will also lack a 
normative base. 



Research and Evaluation Outputs: Schalock et al. (16) identified 
three types of outputs: conditions, events, and products. 
Conditions (circumstances) and events (transactions or behaviors) 
are hard to observe unless the individual being assessed is already 
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in one's employ* The evaluation of products (such as research 
articles and evaluation reports) has been the focus of much writing* 



Bartos (2) provided a bibliography of several dozen checklists 
for evaluating research articles* Stufflebeam et al* (19) proposed 
the following criteria to judge evaluation work: internal validity, 
external validity, reliability, objectivity, relevance, importance, 
scope, credibility, timeliness, pervasiveness, and efficiency (later 
named cost effectiveness)* The work on m .ta-^evaluation (that is, 
evaluating evaluations) is relevant here, and perhaps ^he most 
detailed statement is provided by Stufflebeam (21)* 

As pointed out by Millman and Gowin (13), a difficulty with 
such lists is they do not provide the criteria to judge the 
criteria* How d 'cs one decide, for example, if research is 
significant or ti 3 measuring devices used in an evaluation are 
valid? Although guidance in appraising research and evaluation 
articles is provided in the Millman and Gowin book, it is perhaps an 
insufficient tool for the busy administrator who, without special 
training, must immediately judge the quality of an applicant's work* 
The administrator might better consider sending samples of the work 
to noninvolved critics* 



Despite the obvious appeal of R and E products as a source of 
evidence about an individual's competence, they do have their 
drawbacks* Reviewers should remember that applicants do not have 
complete freedom In the selection and conduct of their work* 
Further, if a report is produced by many people, it is most 
difficult to untangle the contributions made by each* Judgments of 
the quality of R and E products should be tempered by these 
cons iderat ions* 

Other Sources: Bibliographic, academic, and other characteristics ^ 
of the applicant such as experience, sex, or type of doctorate 
degree earned, have been correlated with subsequent production 
(usually in a research setting)* The results of these studies do 
not make one hopeful of finding bibliographic and academic 
characteristics vlth a strong enough relation to amount or quality 
of R or E to be taken seriously as selection variables* (For a 
review of studies in this area, see 17*) 

In spite of a lack of positive empirical support, 
recommendations have often been used. In view of recent regulations 
on accessibility of records, administrators are cautioned to have 
direct contact with individuals best able to judge the promise of 
the applicant. These individuals may not be the ones the applicant 
has chosen to reference* 



Finally, there has been a serious suggestion that the 
qualifications of professionals in educational research and 
development be judged by their membership in special professional 
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associations and directories having more stringent entry 
requirements than presently exist (?)• 



This last suggestion points up once again the need for some 
certifying mechanism to protect both the consumer and the 
profession. Like a physician, the researcher and evaluator are 
involved in work that can have serious consequences. Moreover, like 
the physician, perhaps the researcher and evaluator should have 
available to them certifying tests similar to medical board 
specialty examinations. Such tests should require the worker to 
perform tasks similar to those found on the job* In the absence of 
certifying procedures, ti.e administrator has no easy way to assess 
educational researchers and evaluators. 
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